texts: Are batch iteration and epochs different in reinforcement learning compared to supervised learning?

texts:
I'm following the Udacity "AWS DeepRacer" course about self driving cars with reinforcement learning.
In one lesson, they says this:

Batch size - This determines how many images, randomly sampled from the most recent episode, are used for training before updating the model.

Epochs - Determines how many times you will loop through the batched data before updating the training weights. A larger number of epochs is likely needed if your model is still improving but training has otherwise ceased.

Learning rate - Controls the speed at which your algorithm learns (it enlarges or shrinks the weight update after each epoch).

Using Tensorflow and Keras with Adam, I learned that the weights are updated after each batch iteration.
But in this course, they says that the weights are updated only after each epoch, and the batch size determine how much of the training data to loop through before updating the model. So, what's the difference between updating the model and updating the weights?
In tensorflow with keras, the weights are updated after each batch iteration and also after each epoch? Is it different compared to RL?


texts: Keras stateful LSTM returns NaN for validation loss

texts:
I'm having some trouble interpreting what's going on in the training and validation loss, sensitivity, and specificity for my model. My validation sensitivity and specificity and loss are NaN, and I'm trying to diagnose why.

My training set has 50 examples of time series with 24 time steps each, and 500 binary labels (shape: (50, 24, 500)). My validation set has shape (12, 24, 500). Of course, I expect a neural network to overfit massively.

Because I was interested in a stateful LSTM, I followed philipperemy's advice and used model.train_on_batch with batch_size = 1. I had multiple inputs: one called seq_model_in that is a time series, and one called feat_in that is not a time series (and so is concatenated in to the model after the LSTM but before the classification step).

As my classes are highly imbalanced, I also used Keras's class_weights function. To make this function work in a multi-label setting, I concatenated two columns to the front of my responses (one of all 0s and one of all 1s), such that the final shape of the response is (50, 502).

 feat_in = Input(shape=(1,), batch_shape=(1, 500), name='feat_in')
 feat_dense = Dense(out_dim, name='feat_dense')(feat_in)
 seq_model_in = Input(shape=(1,), batch_shape=(1, 1, 500), name='seq_model_in')
 lstm_layer = LSTM(10, batch_input_shape=(1, 1, 500), stateful=stateful)(seq_model_in)
 merged_after_lstm = Concatenate(axis=-1)([lstm_layer, feat_dense])
 dense_merged = Dense(502, activation="sigmoid")(merged_after_lstm)

I have coded this model in Keras for time series prediction (multi-label prediction at the next time step):

The training and validation metrics and loss do not change per epoch, which is worrisome (and, I think, a symptom of overfitting), but I'm also concerned about understanding the graphs themselves.

Here are the TensorBoard graphs:

The training loss should (roughly) be decreasing per epoch, as should the validation loss. The training sensitivity and specificity are 92% and 97.5%, respectively (another possible sign of overfitting).

My questions are:

Am I right in thinking the sensitivity and specificity graphs should all have the same general shape as the train_spec graph?
Why does the training sensitivity graph look like this?
Why does the validation loss return NaN?


texts: How to validate different method of clustering

texts:
I have applied K-means and Hierarchical Agglomerative clustering method on some data and clustered them into 5 groups. For validation (agreement) purpose I used this formula:

c=(# of cluster shared/ Total cluster)*100 %

I feel its a wrong validation technique.

Can I use entropy, Rand index, Dunn number etc to measure the validation?


texts: Same accuracy on KNN for different distance metrics

texts:
I am new to data analysis and I am trying to run a kNN classifier on a lung cancer dataset with multiple attributes. For all k values I tasted (1 to 10), I obtain the same accuracy when using either Manhattan or Euclidean distances. I am not sure what this is saying about the data? Is the reason that there's no difference the fact that I don't use a large enough data set (300 entries with a 70% training split)?
Thanks!


texts: How Natural language processing and elasticsearch are related

texts: I got call from a recruiter for an elasticsearch job. I am not sure how these 2 are related. Can anyone suggest and provide example use case?


texts:
I'm trying to explore an use-case in ML but stuck at a point. May i please request your advise please.

Have a service desk web application for logging tickets, which is essentially a form having various fields like -

subject,
content,
username,
emailid,
category - {hardware, application, datafix, mobileapp, etc.},
service group - {AAA, BBB, CCC, DDD, etc.},
domain - {email, walkin, phonecall, etc.},
priority - {high, medium, low}
#(Apologies for the poor quality of sample data provided above.)

Based on this info, the ticket is then manually assigned to respective team owners for resolution.

My intent is to use ML - based on the above fields, predict the Team who will work on this ticket. (Team ex. HR or IT or Desktop support or Pantry or Facilities, etc. )

Can this use-case be categorized as Multi-class classification
problem?
The field values are stored as words in database. How can it be fed
to my ML as numbers?

Please advise.


texts: GridSearchCV results are different to directly applied default model (SVM)

texts:
I run a Support Vector Machines model on part of my train set with following result:

alg = sk.svm.SVC(probability=True, gamma='auto')
cv_results = model_selection.cross_validate(alg, X_pca, labels, cv =4)

but when I am trying to tune the parameters, with following method:

model=sk.svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0.1, 1, 'auto'],
      'probability' : [True]
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False).fit(X_pca, labels)
pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score', 'params']].sort_values(by='rank_test_score')

So not only all results looks scetchy because they are the same. but also in one of the rows I have C:1, gamma:auto and probability: True which is the same parameters as in first table.

I want to also say, that the same logic I am using for the rest of my 15 ML algorithms and only SVM showed this kind of weird behavior. Wondering that maybe I have some stupid mistake in how I create X_pca and labels data table, I copied code from other algorithm and just replaced second code but it gave the same results.

Can you spot something wrong?


texts: Entity recognition with context/relation

texts:
Is there a way to get a specific entity based on the context where it is found?
For example:

The temperature today is 35°C.

Store risperidone tablet at 20°C.

Both are talking about temperature. For the first sentence I would want the temperature to be "WeatherTemperature" entity. The second sentence I would want the temperature to be "DrugTemperature". What model could I use to train for this behavior?


texts: Class weight degrades Multi Label Classification Performance

texts:
I noticed something strange while I was conducting a multiple label classification problem via keras neural network. My data set consist of imbalance data with 12 features and 25 possible labels. When I instantiate my model with no class weight I get a precision of 97%, recall of 13%, subset accuracy of 14%, f1-score of 23% using the micro average.
When I apply class weight these scores are significantly reduced to the below.

('Accuracy', 0.1757093081134893)
('Precision:', 0.19632925472747498)
('Recall', 0.1637291280148423)
F1 -score 0.178553363682

Also I calculate the weights with below code that I copied and modify from a previous post:

def class_out(s):
      y_classes = s#.idxmax(1, skipna=False)

      # Instantiate the label encoder
      le = LabelEncoder()

      # Fit the label encoder to our label series
      le.fit(list(y_classes))

      # Create integer based labels Series
      y_integers = le.transform(list(y_classes))

      #print y_integers
      # Create dict of labels : integer representation
      labels_and_integers = dict(zip(y_classes, y_integers))

      print labels_and_integers
      class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
      sample_weights = compute_sample_weight('balanced', y_integers)

      class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
      class_sweights_dict = dict(zip(le.transform(list(le.classes_)), sample_weights))

      print class_weights_dict

      return class_weights_dict

Also see a sample of the model:

batch_size = 100

weights = class_out(df_all['tag'])

model = Sequential()
model.add(Dense(10, activation="relu", input_shape=(12,)))

#model.add(Dense(10, activation='relu'))

#model.add(Dense(8, activation='relu'))
#model.add(Dropout(0.50))
model.add(Dense(25, activation="sigmoid"))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy',precision,mcor,recall, f1])

model.fit(X_train, Y_train, batch_size=batch_size, epochs=15,class_weight=weights,
      verbose=1,validation_data=(test, target_test))

Is there a reason to believe that the model performance is best without class weights ?


texts:
I have ensembled 3 algorithms as below,

estimators = []
model1 = MultinomialNB().fit(X,y)
estimators.append(('Naive Bayes', model1))
model2 = LinearSVC(random_state=0).fit(X,y)
estimators.append(('SVM', model2))
model3 = RandomForestClassifier(bootstrap=True).fit(X,y)
estimators.append(('Random Forest', model3))

# create the ensemble model
ensemble = VotingClassifier(estimators)
results = model_selection.cross_val_score(ensemble, X, y)
print(results.mean())

while passing new sentence to test the ensembling I have used,

predicted = results.predict(X_new_tfidf)

But I am getting error:

  AttributeError: 'numpy.ndarray' object has no attribute 'predict'

How can I fix/debug this?


texts: Finding both count and average of a column in R data.table, after group by

texts:
I have a data table of 5 columns. I want to do a group by on one column  and want to get the count of another column, id.

I do this in this way:  dt[,length(id),by=dt$cid].

If I also want to get sum of id, how do I proceed?
I tried this:

dt[,(sum(id) length(id)),by=dt$cid]

but it did not work. Could you please help out.

Thanks


texts: Import the same interval of previous week into the deep model

texts:
In a dataset, the data are the average of vehicles speed in the points (cells) of a map. I am trying to build a prediction model.

While the inputs are the average of vehicles speed of all points in recent 40 min and the outputs are the next 10 min of all points.

If there were 320 points on the map, the input shape = (320,20) and the output shape =(320,5). The averages of vehicles speed are collected each 2 min in all points.

There are several such samples (n samples, x=(n,320,20), y=(n,320,5)).

The code of the final architecture of my model:

model = models.Sequential()
model.add(Conv2D(256,(3, 3),
         activation='relu',
         input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
model.add(Dense(1600))

model.summary()

There is a significant weekly cycle and I would like to import the same interval of the previous week together with the current input to the model.

How should I change the model?

Should I change the input shape to (n, 320, 2 * 20)? adding additional average speeds from previous week.


texts: regarding lasso.score in lasso modeling using scikit-learn

texts:
I once saw the following code segment of using lasso model based on scikit-learn
lasso = LassoCV(normalize = True, max_iter=2000)
lasso.fit(X_train, y_train);
lasso.score(X_test, y_test)

y_pred = lasso.predict(X_test)

I can understand lasso.fit and lasso_predict, but what does lasso.score generally offer? According to the scikit-learn, it Returns the coefficient of determination R^2 of the prediction. I am not quite clear how to use this kind of information?


texts: Extracting component means and convariances from mixture model

texts:
I am currently trying to write a simple multivariate gaussian mixture model using tensorflow probability. Specifically, I have some 2-dimensional input and 2-dimensional output data and am looking to produce a probabilistic model that best fits this data by using a neural network. I am utilizing tfp.layers.MixtureSameFamily to generate this model, which is working perfectly well as expected. Here is my code for doing so:
Notes: I am using tensorflow 2.4.1 and tensorflow-probability 0.12.1
x = some 2-d input data
y = some 2-d output data

event_shape = y.shape[-1]
num_components = 5
params_size = tfp.layers.MixtureSameFamily.params_size(
num_components,
component_params_size=tfp.layers.MultivariateNormalTriL.params_size(event_shape))

model = tf.keras.Sequential([
tf.keras.layers.Dense(params_size),
tfp.layers.MixtureSameFamily(num_components, tfp.layers.MultivariateNormalTriL(event_shape)),
])

model.compile(optimizer=tf.optimizers.Adam(learning_rate=0.01),
loss=lambda y, model1: -model1.log_prob(y),
metrics=[])
batch_size = 250
model.fit(x, y,
batch_size=batch_size,
epochs=500,
steps_per_epoch=n // batch_size,
verbose=False,
shuffle=True)

yhat_mix=model(x_tst)

However, once the network has been trained and the model produced, I do not know how to extract the specific information that I am actually after, i.e. the means and covariances of the component 2-dimensional gaussians that make up the mixture model. Is there an easy method to extract this information? I've been looking for something akin to ".mean()" and ".covariance()" but which give the individual component means and covariances, instead of for the entire distribution. I thought using ".mode()" might help since I'm partly looking for the modes of the entire distribution but this just returns the following error:
NotImplementedError Traceback (most recent call last)
in
----> 1 yhat_mix.mode()

~\anaconda3\envs\py36\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in mode(self, name, **kwargs)
1394 """Mode."""
1395 with self._name_and_control_scope(name):
-> 1396 return self._mode(**kwargs)
1397
1398 def _cross_entropy(self, other):

~\anaconda3\envs\py36\lib\site-packages\tensorflow_probability\python\distributions\distribution.py in _mode(self, **kwargs)
1389 def _mode(self, **kwargs):
1390 raise NotImplementedError('mode is not implemented: {}'.format(
-> 1391 type(self).name))
1392
1393 def mode(self, name='mode', **kwargs):

NotImplementedError: mode is not implemented: MixtureSameFamily

Ideally, I can get the parameters that define all the component multivariate distributions though (their means and covariances) and not just mode locations of the mixture model. Does anyone know how to get this information? Any help would be appreciated, thanks!


texts:
I'm reading Cliche, Mathieu (2017). In his paper, he describes using cross entropy loss, weighted by the inverse frequency of the true classes to counteract an imbalanced dataset (this is stated on page 5, first paragraph).

Mathematically, what does this mean?

I know that cross entropy loss is defined as:

$$
-\sum_iy_i\log(\hat{y}_i)
$$

where $i$ is a particular node in the output layer, $y$ is the true distribution and $\hat{y}$ is our distribution in the output layer.

How is this weighted by the inverse class frequency?


texts: Can Generative Adversarial Network be run on any embedded / edge device?

texts:
I am using DCGAN ( Deep Convolution GAN ) to generate images. However, I want to run it on embedded devices, such as Intel Movidiusor Jetson.

But, while going through several internet articles, I came to know that embedded devices are only used for inference in case of deep learning.

I have a few questions :

GAN is basically a generative network. How can it have inference ? Will the inference and training be same thing for GAN ?
Can it (GAN) be run on embedded devices , where memory is really an issue?


texts:
In this code below, a picture can be loaded into openCV and then the region of interest RIO can be created by just selecting a box around something with the mouse, then press enter.

What I am trying to do is measure the apparent width in pixels of the ROI. I am going thru some of the theory in this pyimagesearch blog post for calculating distances in images.

I have a folder called images and a .png file named 1 within the folder.

import cv2

image = cv2.imread("images/1.png")
cv2.imshow("image", image)
marker = cv2.selectROI("image", image, fromCenter=False, showCrosshair=True)

print(marker)

print(marker[1]-marker[3])

The output is below but I am not really sure how/if this can be used. I think its stating the ROI x,y height & width but I am not sure:

(317, 181, 269, 359)
-178
>>>

Any tips help... Thanks


texts: Can I use more features for my training data than my test data will supply?

texts:
I am pretty new to the data science game so pardon me, if the answer to my question should be a no-brainer.

We are looking at manufacturing / quality data where products are labeled 'okay' or 'not okay' based on different measurements. In our data set, we also have information on the product type, which machine was used to build it, at what time it was build, temperature, etc.

We are trying to train a classification model that will correctly classify 'okay' or 'not okay' products. At the point where the classification should be done, we do not have the measurements of the product, but all the other data I mentioned. Someone suggested using all data to train the model incl. the measurements, even though the test data will not have this information.

So the question is: is it feasible and does it make sense to train a model with more features than it can test on the test data? What ever the answer - please explain because I just can't seem to wrap my head around it.

Kind regards

Julia


texts: (Almost) linearely separable dataset, where can I find one?

texts:
I'm implementing the perceptron algorithm and the voted perceptron algorithm for an assignment for university.
For that I need to find some decent datasets.. I've tried the UCI repos and I've come up with:

The banknote authentication dataset:
https://archive.ics.uci.edu/ml/datasets/banknote+authentication
Occupancy dataset: https://archive.ics.uci.edu/ml/datasets/Occupancy+Detection+

that work quite nicely, the accuracy is pretty high for the linear separability.
HOWEVER I need to find a third dataset to end my assignment, I've been searching everywhere but I can't seem to find anymore of them, the dataset has to have more than 1000 examples, and it has to be binary classifiable obviously..

Does anyone of you know where I could find such a dataset?

I've been trying the Adult dataset, but it's pretty much sparse.. far from being linearly separable..

NOTE: I can't use the kernel trick or any other higher dimensional paradigm.


texts: Should one-hot encoded categorical features needs to be scaled when used along with text feature while deriving semantic similarity?

texts:
My aim is to derive textual similarity using multiple features. Some of the features are textual for which I am using (Tfhub 2.0) Universal Sentence encoder. There are other categorical features which are encoded using one-hot encoder.
For example, for a single record in my dataset, feature vector looks like this:

text feature's embedding is 512 dimension vector - 1 X 512
categorical (non-ordered) feature vector -  1 X 500 (since there are 500 unique values in the feature)
my final feature vector - 1 X 1012

After this, I derive similarity matrix using cosine-similarity to decide if two such records are semantically same or not.
Problem is, there is a difference in the range of values for text feature (real numbers) and one hot encoded feature (0 or 1).
So shall I scale the one hot encoded vector with min-max scalar or using some other technique?


texts: Multilayer Perceptron: What is the value used to update the weights in the hidden layers?

texts:
As i understand for the output layer the error rate is used with the mean squared error function to update the weights.
For the hidden layers as well? Does that make sense?


texts:
I am running Gradient Checking, to spot any discrepancy between my mathematically-computed gradient and the actual sampled gradient - to reassure my backprop was implemented correctly.

When computing such a discrepancy, can I sum-up squares of differences, then take their average? I could then use this average as my estimate of how correctly the network computes the gradient:

$$\frac{1}{m}\sum_{i=0}^{i=m}(g_i-n_i)^2$$

or even:

$$\sqrt{\sum_{i=0}^{i=m}(g_i-n_i)^2}$$

where $g$ is a gradient from backpropagation, and $n$ is gradient from gradient checking.

However, Andrew Ng instead recommends:

$$\frac{\vert \vert (g-n) \vert \vert _2 }{
\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$

where $\vert \vert . \vert \vert _2$ is the length of the vector.

Another post post also recommends an slightly different approach: https://stats.stackexchange.com/a/188724/187816

Why would their approaches be better than mine?


texts: Linear Regression Error in feature matrix step

texts:
I'm trying to code the design function used in linear regression using numpy and I get this error:

  Traceback (most recent call last):
  File "C:\Users\visha\AppData\Local\Continuum\anaconda3\lib\site-
  packages\nose\case.py", line 197, in runTest
     self.test(*self.arg)
  File "C:\Users\visha\machinelearning\test.py", line 23, in test_compute_Phi
   assert type(Phi) == np.matrixlib.defmatrix.matrix
  AssertionError

input > x: input vector of shape n x 1,
         p: the number of polynomials/features

TEST CASE :   x = np.mat('1.;2.;3') , P=2

can you please help me correct this code ?

code

def compute_Phi(x,p):

    x = np.asarray(x)
    Phi = np.zeros(shape=(len(x),p+1))
    for i in range(0,p+1):
        Phi[:,i] = np.power(x,i).reshape(x.shape[0],)

    return Phi


texts: Link Prediction in Directed Social Network using Supervised Learning

texts:
I am trying to solve the link prediction problem in a directed graph using supervised learning. In the case of an undirected graph, it is pretty straightforward. For instance, first, compute the embeddings and solve the problem as a via binary classification problem, where, node embeddings need to be transformed into edge embeddings by averaging or adding ‘i’ and ‘j’ node embeddings. This makes sense in an undirected graph because i ->j and j->i edges represent the same link. However, in a directed graph, it doesn’t make sense to transform the node embedding pair into edge embedding by adding or average. Can anyone help me out on how I can convert the node embedding into edge embedding?
Looking forward !


texts: Which solutions are there for RL agents when not all actions are always available?

texts:
I'm working in an RL environment where not all actions are always available. In this case, depending on the state where the environment is at, some of the actions are not available for the agent to choose.
I could find this work that deals with this situation: https://ojs.aaai.org/index.php/AAAI/article/view/5740
Before moving on and starting implementing it, I'd like to ask if you have any other suggestions of solutions to deal with such a scenario.
I'd appreciate any pointer.


texts:
When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where $x$ is an input vector and $\varepsilon$ is the random error term.

In the case of a Prediction algorithm, this is the equation :

$\hat{y}=\hat{f}(x)$ where $\hat{f}$ represents our estimate for $f$, and $\hat{y}$ represents the resulting prediction for $y$.

What I don't understand is : why n this setting, $\hat{f}$ is often treated as a black box, in the sense that one is not typically concerned with the exact form of $\hat{f}$, provided that it yields accurate predictions for Y.


texts: Events prediction with time series of continuous variables as features

texts:
We have the feeling that behavior of a device in terms of continuous variables (fans speeds, temperatures, voltages, ...) has influence on rare events happening (components failures).

I now have to build a predictive model for that, to proof the influence.

Those continuous features are given as time series, and events are punctual.

I've made a model based on descriptive statistics of those variable (see this question) with decision tree, random forest, adaboost, and clustering but it doesn't works. I will still improve by balancing classes, but I'm convinced it is not the best approach.

I'm pretty sure that there are nicer algorithms for such predictions (this is quite common problem), but I don't find anything.

Do you have ideas?

Thanks a lot

PS: I'm working with Python and cython


texts: How can I improve my model on a very very small dataset?

texts:
I am starting as a PhD student and we want to find appropriate materials (with certain qualities) from basic chemical properties like charge, etc. There are a lot of models and datasets in similar works, but since our work is pretty novel, we have to make and test each data sample ourselves. This makes the data acquisition very very slow and very expensive. Our estimated samples will be 10-15 samples for some time, until we can expand it.

Now I want to use this samples to make a basic predictive model, but with as much 'good generalization' as possible. I will use this model to screen other possible candidates from a large pool of properties to find best probable materials, and will then proceed to make them for testing.

Now I clearly don't expect performances anywhere near 95% or so, but I want a working model with predictive capability that will actually help me to find some of the best probable material candidates, so we can expand our work. I am uncertain if I can (or rather should) use some of regular ML methods like dataset splitting and cross-validation. So I would appreciate your thoughts.

Since our data size is minuscule, I have been searching for ways to improve its robustness. These our my ideas:

1- Use an ensemble model to avoid overfitting and avoid skewed biases (using algos like elasticnet, SVM, random forests, etc).

2- Setting heavy regularization to avoid certain biases that can arise from small data.

3- Using algos that arrive at the minimum periphery faster.

I would appreciate any suggestions on how I can improve this model as much as possible, to reach the best generalization performance.

I have also thought about synthetic data generation a lot. Do you have any suggestions on how I can go about it?


texts: What Introductory Statistics book would you recommend?

texts: Is there a particularly good book or good publisher for teaching Statistics from beginner level for someone interested in studying Data Science?


texts:
I'd import 'tweepy' package for accessing twitter data for analysis and visualization in Python. Afterwards I did setup oauth authentication and everything else but on the next step, which is STREAMING TWEETS for me am using MyStreamListener. This causing me a Name Error--

  NameError: name 'MyStreamListener' is not defined

Here's my code-

# API Authentication

# Importing TWEEPY Library
import tweepy

# Store OAuth authentication credentials in relevant variables
access_token = "provided"
access_token_secret = "provided"
consumer_key = "provided"
consumer_secret = "provided"

# Pass OAuth details to tweepy's OAuth handler
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Streaming Tweets

# Initialize Stream listener
l = MyStreamListener()

# Create you Stream object with authentication
stream = tweepy.Stream(auth, l)

# Filter Twitter Streams to capture data by the keywords:
stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])

What should I do now? Any thoughts?
Cheers!


texts: Need rules of thumb for out of core larger than ram dataset on a laptop

texts:
totally new to larger than ram datasets but I have csv files that are about 100 gb each with around 300 million rows each  (just two of them).

What I am looking for is something like: don't go analyzing more than 1 terabyte of data on a laptop. Or don't analyze data that is more than 10x your ram or the wait times will prove frustrating. This is what I mean by rules of thumb.

I have a mac laptop with an i5, 8gb ram, ssd is it reasonable to process this data (100gb per file and I have 2 files) with either dask or blaze in python?

I have tried and could read in the csv but when doing simple trials like dropping one column or finding the length of the data frame it takes at least an hour For the drop (I gave up waiting) and half an hour for the length() (that did finish). Way too slow to be productive. I changed the data format to parquet and amazingly the data is down to just 10 gb. This is the only way they the length() command ever even finished.

So my question is: are these response times normal experience in general given my hardware and csv size? Are my expectations for what dask can do way too high? Any rough guidelines on how to troubleshoot or is this clearly underpowered hardware？


texts:
I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a validation split or a subset in a code? or I can use the whole files as train_ds and val_ds by deleting them
Folder names in the training and validation directory are same.
data_dir = 'trainingdatav1'
data_val = 'Validationv1'

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
                                                                data_dir,
                                                                validation_split=0.1, #is it required if I'm gonna use the whole folders and files for training?
                                                                subset="training",
                                                                seed=123,
                                                                image_size=(img_height, img_width),
                                                                batch_size=batch_size)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
                                                              data_val,
                                                              validation_split=0.8, #need to check
                                                              subset="validation",
                                                              seed=455,
                                                              image_size=(img_height, img_width),
                                                              batch_size=batch_size)

num_classes = 14

model = tf.keras.Sequential([
  layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)),

  layers.Conv2D(16, 3, padding='same', activation='softmax'),
  layers.MaxPooling2D(),

  layers.Conv2D(32, 3, padding='same', activation='relu'),  #from renu
  layers.MaxPooling2D(),

  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(.2),             #prevent overfitting

  layers.Flatten(),
  layers.Dense(128, activation='sigmoid'),
  layers.Dense(num_classes)
])

model.compile(optimizer='SGD', #adam
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
model.summary()

epochs=50
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

Another question is the overfitting issue - validation accuracy is not over 0.4 and val_loss is around 2.xxx. Suggestions from Stacexchange are:

Reduce the layers of the neural network.
Reduce the number of neurons in each layer of the network to reduce the number of parameters.
Add dropout and tune its rate.
Use L2 normalisation on the parameter weights and tune the lambda value.
If possible add more data for training.

Are there any other suggestions?


texts: Does it matter whether we put regularization parameter ($C$) with error or weight term in Kernel ridge regression?

texts:
Kernel ridge regression associate a regularization parameter $C$ with weight term ($\beta$):

$\text{Minimize}: {KRR}=C\frac{1}{2} \left \|\beta\right\|^{2} + \frac{1}{2}\sum_{i=1}^{\mathcal{N}}\left\|e_i  \right \|_2^{2}  \\
 \text{Subject to}:\ {\beta^T\phi_i}=y_i - e_i, \text{  }i=1,2,...,\mathcal{N}$

If we associate $C$ with an error term as follows:

$\text{Minimize}: {KRR}=\frac{1}{2} \left \|\beta\right\|^{2} + C\frac{1}{2}\sum_{i=1}^{\mathcal{N}}\left\|e_i  \right \|_2^{2}  \\
 \text{Subject to}:\ {\beta^T\phi_i}=y_i - e_i, \text{  }i=1,2,...,\mathcal{N}$

then how this second formulation is different from the first one?

or

Can we associate $C$ either with weight term or error term in Kernel ridge regression?


texts:
I am a beginner and need further help to elaborate following problem. I want to "read" text from images. In my problem setting, the text on all the images has the same structure. I found this solution by googling about "optical character recognition".
The link mentions that I need an object localization algorithm. How can I implement this in tensorflow?
Thank you!


texts: Calculating image text orientation in scanned documents

texts:
I'm trying to calculate the text orientation in scanned documents in order to de skew them. I'm aware of some of the common methods which include using hugh transform to do the same. However I recently read an article which I can't find.
If I recollect correctly, it basically said that find the nearest neighbours for the connected components in the image. This should connect the adjacent alphabets in the words, connect the centroid of those components and build a histogram of those vectors. The histogram should give you the skew angle of the vectors.
Does anybody know such an approach for identifying document skew?
Any other approaches are appreciated as well.
Thank you.


texts: What is the use of user data collection besides serving ads?

texts:
Well this looks like the most suited place for this question.

Every website collect data of the user, some just for usability and personalization, but the majority like social networks track every move on the web, some free apps on your phone scan text messages, call history and so on.

All this data siphoning is just for selling your profile for advertisers?


texts: Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

texts:
I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here.
I first chose columns to work with:
    df_euro = pd.read_csv('https://assets.datacamp.com/production/repositories/655/datasets/2a1f3ab7bcc76eef1b8e1eb29afbd54c4ebf86f2/eurovision-2016.csv')
   samples = df_euro.iloc[:, 2:7].values[:42]
   country_names = df_euro.iloc[:, 1].values[:42]

        # Calculate the linkage: mergings
            mergings = linkage(samples , method = 'complete')
            # Plot the dendrogram
            dendrogram(
                mergings,
                labels = y,
                leaf_rotation = 90,
                leaf_font_size = 6
            )
            plt.show()

But I'm getting this error which I can't understand. I googled it and checked that both have the same shape (1066,5) and (1066,). There is no NA in both features and country_names.
I know the issue is with labels because it works well when I remove it but I couldn't find away to solve it. Any help will be really appreciated. Alternately, how can I add the labels to the Dendrogram?
The error I am getting:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-113-7fffdc847e5e> in <module>
      4 mergings = linkage(feat, method = 'complete')
      5 # Plot the dendrogram
----> 6 dendrogram(
      7     mergings,
      8     labels = target_col,

C:\ProgramData\Anaconda3\lib\site-packages\scipy\cluster\hierarchy.py in dendrogram(Z, p, truncate_mode, color_threshold, get_leaves, orientation, labels, count_sort, distance_sort, show_leaf_counts, no_plot, no_labels, leaf_font_size, leaf_rotation, leaf_label_func, show_contracted, link_color_func, ax, above_threshold_color)
   3275                          "'bottom', or 'right'")
   3276
-> 3277     if labels and Z.shape[0] + 1 != len(labels):
   3278         raise ValueError("Dimensions of Z and labels must be consistent.")
   3279

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py in __nonzero__(self)
   1476
   1477     def __nonzero__(self):
-> 1478         raise ValueError(
   1479             f"The truth value of a {type(self).__name__} is ambiguous. "
   1480             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().


texts: Self Organising Map with variable length ordered sets of N-grams

texts:
I want to preface my question with the highlighted situation I have might not be applicable to kohonen self organising maps (SOM) due to a lack of understanding on my part so I do apologise if that is the case. If this the case I would greatly appreciate any suggestions on alternative methods to compare the similarities for my given input data.
I am trying to create a self organising map for the similarity comparison between the ngram ordered set of tokens  that can be extracted from a given input sequence. For example, if the tokens are:
[A, B, C]
And my input is:
A, B, B, C, A
Then the resulting 1-gram ordered set would be:
A, B, C
I will try to highlight my question using a simple example followed by some more realistic cases I am dealing with.
If I have the collection of tokens:
[A, B, C, D, E, F]
The input data contains K variable length strings, made from these tokens.
A, B, A, B, C, D
A, A, A, B, B, E, F
A, B, C
E, F, E, E, F, D, C, A, A, A, A

I then generate the unique ordered sets for the tokens found in each of these sequences using the alphabetical order of  tokens.
A, B, C, D
A, B, E, F
A, B, C
A, C, D, E, F

The above would be the 1-gram ordered set for the values. I however need to be able to represent the N-gram ordered sets.
E.g. for 2-grams the ordered set of tokens for each sequence would be:
AB, BA, BC, CD
AA, AB, BB, BE, EF
AB, BC
AA, CA, DC, EE, EF, FD, FE

And for 3-grams:
ABA, ABC, BAB, BCD
AAA, AAB, ABB, BBE, BEF
ABC
AAA, CAA, DCA, EEF, EFD, EFE, FDC, FEE

The 4-grams would only produce 3 ordered sets rather than 4 as the sequence A,B,C only has 3 tokens.
My initial thought was to have the of attributes be all the permutations of these n-gram token sequences.
For 1-grams this would:
A B C D E F
And for 2-grams this would be:
AA AB AC AD AE AF BA BB BC BD BE BF CA CB CC CD CE CF DA DB DC DD DE DF EA EB EC ED EE EF FA FB FC FD FE FF
Unfortunately, this very quickly gets out of hand, even when just going from 1-grams to 2-grams. With the number of attributes increasing by T^N... This seems completely impractical and illogical especially given the majority of the resulting matrix consist mainly of 0's where the attribute doesnt exist in a given sequence.
The problem is only made worse as the collection of tokens I have is closer to 150 rather than the 6 used in this example.
Each possible ordered set is also highly variable in length and will range from 1 to the number of permutations for the given token set.
The other issue is even if I was to optimise the number of attributes to only include the existing permutations in the sequence, there is no guarantee that testing data will conform to this set of attributes.
I am guessing that I will need to somehow convert these ordered sets to a more manageable set of attributes, but I am unsure of what would be a viable solution as each set of tokens seems to have an equal importance weighting.
I am also assuming that this be difficult to generalise and each N would have to have a different trained model? (One for 1-grams, another for 2-grams... up to the maximum N-gram chosen?)


texts:
I need to detect the rotation of a cable (degree) in the x-axis with high precision [0.2 (or more) degree detection] from its original state.
Detailed description:

I have a cable that is set in its original state.
The system has rotated the cable in the x-axis.
I want to know the degree the cable has been rotated from its original state.

Example:
There're following images for a specific cable in different rotation (angle) [0, 0.4, 0.6, 0.8]:
1)
2)
3)
4)
[First image shows the initial state of the cable; the following three images show 0.4, 0.6, 0.8 rotation from its initial state respectively)].
Important facts:

It is not possible to make pictures in better resolution.
It is not possible to make pictures in better contrast.
The cables may vary (different positions of the light strips on the cable).

Can this problem be solved using computer vision, neural networks, or other methods/techniques?


texts: Does anyone have suggestions for how Goodfellow et al detected digits from the larger Street View images in their 2014 paper?

texts:
In their 2014 paper titled, "Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks", they write,

  We use an automated method (beyond the scope of this paper) to
  estimate the centroid of each house number, then crop to a 128 × 128
  pixel region surrounding the house number.

Can anyone suggest possible methods they used to accomplish this task?


texts: What is neural structure learning in tensorflow?

texts:
What is neural structure learning?
what is the difference between neural network vs neural structure
learning?


texts: How to do time-series prediction in the browser

texts:
I've been trying to find a simple tutorial to take me through using TensorFlow.js to do predictions on a series of values, but I haven't been able to find something simple and based in JS.

I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.  I have over a million values in this set, and they're just simple values in sequence.

Any direction or helpful and simple tutorials would be really appreciated.


texts: What is the meaning of the Variant Q-learning and To what INPUT and OUTPUT refer? in Abstract of DeepMind DQN paper 2013

texts:
-INPUT and OUTPUT OF ATARI DQN:
In the abstract paragraph of the DQN work by DeepMind it has written:
" We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. "
Would you please explain the bold sections?
•   Input and output refer to Q-learning or DNN?
•   Why it is said that learn control policy, however DQN is composed of QL which is a value iteration and"off-policy" algorithm and it is not a policy iteration and "on-policy"?
•   What is the meaning of variant type of Q-learning is vague? it is variant because approximation of Q-learning is implemented?


texts: How can I perform categorical encoding when the dataset is too large for memory?

texts:
I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning with either Keras using .fit_generator() or sklearn using .partial_fit(), I'm kinda at a loss as to how to do the categorical encoding steps in such a scenario. Obviously OrdinalEncoder requires a knowledge of every possible value in the feature to fit itself fully...but I can't give it that because I can't load the data into memory.

I haven't tried, but it might be possible to load a single categorical feature at a time and train an encoder on that data, before deleting it from memory and loading the next feature to train another encoder. That seems quite clunky though given there's not actually any reason something like categorical encoder couldn't partially learn categories in an iterative fashion, but I can't find anything in sklearn that will accommodate that.

Is there no "online learning" equivalent for preprocessing data?


texts: Best ML model for predicting yearly data with many blocks?

texts:
I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect.

I would like to use a machine learning model for the current dataset that I have gathered. The dataset has about 8000 blocks (or entities), and each of these blocks has 7 features. Each feature has around 10 yearly values, which are counts (ex: 2005=1, 2006=2 ... 2014=0, 2015=3). The long stretch goal would be able to predict the 11th, and even 12th, yearly value for each feature.

I would really love to apply machine learning to this dataset. I've looked into the following because they seem to be the best solution to my problem: linear regression models, LSTM neural networks, and time series forecasting, although I am still unsure of how my dataset can fit into these certain models.

Please let me know if I can offer any more information!


texts:
I am reading the book The Data Warehouse Lifecycle Toolkit by Ralph Kimball. I come across the term Stovepipes fairly often. After doing some research I read that Stovepipes are when you don't have conformed dimensions to link data marts. This short description was all I could find every other resource I looked at simply said that Stovepipe models are bad.

What is is meant by "conformed dimensions"? What exactly are these Stovepipes and can somebody provide me with an example?


texts: Why is Keras fit_generator different from fit?

texts:
I would like to use a custom generator so that I can implement custom augmentations on my dataset in Keras. However, I built a generic generator (without augmentation) and am confused why it is performing way worse than the built-in fit function.

My generator looks like this:

def data_generator_plain(features, labels, batch_size):

    batch_features = np.zeros((batch_size, IMG_SIZE, IMG_SIZE, NUM_CHANNELS), dtype=np.float64)
    batch_labels = np.zeros((batch_size, NUM_KEYPOINTS * 2), dtype=np.float64)

    while True:
        steps = len(batch_features) // batch_size
        for i in range(steps):
            for j in range(batch_size):
                batch_features[j] = features[(i*batch_size)+j]
                batch_labels[j] = labels[(i*batch_size)+j]

            yield batch_features, batch_labels

and when i call model.fit_generator(data_generator_plain(X_train, y_train, BATCH_SIZE), steps_per_epoch=X_train.shape[0]//BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(X_val, y_val)) the val_accuracy ends around 73% while the basic model.fit(X_train, y_train, epochs=NUM_EPOCHS, batch_size=BATCH_SIZE, validation_data=(X_val, y_val)) val_accuracy ends around 93%.

Is my generator implementation correct? In theory, it should perform the same as the built-in fit function, right? I imagine it is not showing the algorithm all the examples it needs to. Thank you!


texts:
I assume that each person on Facebook is represented as a node (of a Graph) in Facebook, and relationship/friendship between each person(node) is represented as an edge between the involved nodes.

Given that there are millions of people on Facebook, how is the Graph stored?


texts: Do we need to use off-policy methods for policy shaping?

texts: Let's say that there is a reinforcement learning task and an agent in a environment. I want a human teacher to manually modify the policy of the agent (policy shaping) to speed up the learning of the agent. Do I have to use off-policy methods or I can get away with on-policy? Why?


texts: Use deep reinforcement learning with recursive actions?

texts:
Can we use recursive actions in deep reinforcement learning ? If yes how ?

For example, in a reassignment problem, for a task i the neural networks returns a node m, and then there will be some condition : If the node m is busy (If there's already a task j assigned to that node), then the task j will be reassigned to another node and so on until a vacant node will be returned (We suppose that there's always at least one vacant node).


texts:
I am trying to fine-tune BERT-base model for binary text classification using multiple features. 3 text features, 4 categorical features. Text features having more than 500 tokens length, and four categorical features having binary values 0,1. For each categorical features created its corresponding derived feature. for ex. 'Impact_Code' having values 0 / 1, i derived new feature as 'Impact_Code_Derived' with the value 'Impact Code Not Available' for '0', and 'Impact Code Available' for 1. Like this, derived new features for all the 4 categorical features and using text features as it is.
While fine-tuning, i am getting bert embeddings for all the 8 features. i.e., I am taking last-hidden-state of BERT for each feature. The size is (Batch_Size x 512 x 768). I do avg. pooling for each token (1 token x 768 features) then the size become (Batch_Size x 512). and then concatenate all the 8 features avg. pooled output (Batch_size x 512 x 8) then pass it to Fully connected layer with tanh activation. This FC layer output size is 1024. Then this 1024 will be passed as input to another FC layer it will provides 2 outputs.
Since, i could see accuracy around 60%, I am not sure, whether

it is right approach to handle categorical features like this
it is right approach to handle multiple features in bert like this.

Please let me know your valuable suggestions. I searched in net, but couldn't find much clear solutions to clear my doubts.


texts: Using dhmm_em to form the hmm of mfccs' from song clips

texts:
I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (depending on song length).

From what I understand hmm_train cannot be used for sequences of vectors and murphyk's toolbox might work. I tried modifying the demo file as follows to try a single song but I'm getting the error 'Attempted to access obsmat(:,61.6357); index must be a positive integer or logical..'

O = 5;
Q = 10;

v=wavread('We are the champions');
coeff=melfcc(v);
data=coeff';

% initial guess of parameters
prior1 = normalise(rand(Q,1));
transmat1 = mk_stochastic(rand(Q,Q));
obsmat1 = mk_stochastic(rand(Q,O));

% improve guess of parameters using EM
[LL, prior2, transmat2, obsmat2] = dhmm_em(data, prior1, transmat1, obsmat1, 'max_iter', 5);
LL

I haven't worked on this stuff before so I apologize in advance if my question comes across as really basic. It would be great if somebody could tell me where I'm going wrong. Also for multiple clips, should I simply append the matrices generated to the one I'm supplying above? Thanks.

Edit: It turns out that the above hmm can only be applied to discrete values which is possibly why I was getting errors. It seems I have to use another function for 'gaussian output' instead, mhmm_em. I'll post another question to figure out its working.


texts: Transformation of matrix with missing values for hierarchical clustering

texts:
Comparing different variables, I got a matrix with lots of missing values.

How do I have to transform the matrix below for hierarchical clustering?

What I have already tried:

x = read.csv("xyz.csv", sep = ";", row.names = 1, header = TRUE)
x[is.na(x)] = FALSE

# transform to correlation matrix
x = cor(t(x), method = "pearson")

# elbow method
fviz_nbclust(x, FUN = hcut, method = "wss")

# cluster with ward method
hc = hclust(dist(x), method = "ward.D2")

# show clusters
rect.hclust(hc,k=3, border="red")
plot(hc)

This clustering result looks wrong:

dissimilarity matrix

Right keyword for further search "dissimilarity matrix".
Now i tried to transform the matrix into a distance matrix.

x= as.dist(x, diag = TRUE)
hc = hclust(dist(x), method = "ward.D2")
plot(hc)

that´s not a solution yet


texts: Does bias have multiple meanings in Data Science?

texts:
What are the meanings of Bias?
And is Under fitting, which is used in machine learning contexts, the same as "Bias"?

I have faced biased data in sampling in statistics but it seems this is a different thing to bias in learning concepts.

I have heard that some data sets are biased, also have heard the model (for example neural network) has low bias or e.g. 'high bias' problem. Are these uses of bias different?


texts:
i have studied the basic of machine learning ,algebra ,statics and probability.now i want start with deep learning for field of image recognition,classification.
i don't know where to start deep learning ,can anyone suggest me where to start and end like syllabus


texts: In sklearn's classification report, is f1 the best accuracy measure?

texts:
In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model?

             precision    recall  f1-score   support

          0       0.70      0.68      0.69      5007
          1       0.65      0.54      0.59      2270
          2       0.37      0.22      0.28       614
          3       0.74      0.30      0.42       252
          4       0.59      0.42      0.49       262
          5       0.35      0.11      0.17       455
          6       0.34      0.23      0.27       248
          7       0.09      0.05      0.06       133
          8       0.38      0.15      0.21       395
          9       0.43      0.31      0.36       182
         10       0.23      0.12      0.16       230

avg / total       0.60      0.52      0.55     10048

As far as I understand it:

Precision tells us the amount of samples the classifier has correctly
marked as true positive out of all positive results.
Recall tells us about the number of samples the classifier was able
to get correct out of all samples in the set.
F1-score is the harmonic mean of precision and recall.

Maybe I'm misinterpreting the classification_report, but wouldn't the f1-score give the best view on the performance of the classifier?


texts: Neural Network - Adjust number of hidden layers and neurons

texts:
I am using KNIME MLP Neural Network learner (If you are not familiarized with KNIME think of that like a package which implements Neural Network to a set of data).

The thing is you can tune the number of hidden layers, the number of iterations and neurons per layers, so here comes the first question: How to know a priori values for these parameters? For instance, for Random Forest 500 trees is supposed to be a standard to test its performance in the first stages.

Second, I know in general add more layers and/or neurons does not ensure any enhance in the prediction, sometimes even the opposite happen. Someone could give some theoric insights respect this topic; why by adding more layers the prediction might get deteriorated?

Thanks in advance!

Sergi


texts: Industrial application(s) of LDA (latent Dirichlet allocation)?

texts:
LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining.

Question What are the industrial systems using LDA or may be some related models ?  (May be Google/Amazon/ ... ? )

PS

I would be also thankful for comments on:
https://mathoverflow.net/questions/304352/latent-dirichlet-allocation-math-words-digest


texts:
I am trying to plot a CDF of 7 files I have. Each file looks like this:

1
2
1.5
2.3

and so on. The problem is about the sizes of the files, which are:

862M
1,8G
2,4G
18G
2,0G
1,8G

I have put up a simple R script which simply loads the files and plot them. When I generate fake files of smaller size, the script works fine. However, it is running since three days and it hasn't produced any output using the full files.

The script is this:

library(ggplot2)
data <- read.table('file1.csv')
data$g = "G1"
data2 <- read.table('file2.csv')
data2$g = "G2"
data3 <- read.table('file3.csv')
data4$g = "G3"
data4 <- read.table('file4.csv')
data4$g = "G4"
data5 <- read.table('file5.csv')
data5$g = "G5"
data6 <- read.table('file56.csv')
data6$g = "G6"
data7 <- read.table('file7.csv')
data7$g = "G7"

dftotal = rbind(data,data2)
dftotal = rbind(dftotal,data3)
dftotal = rbind(dftotal,data4)
dftotal = rbind(dftotal,data5)
dftotal = rbind(dftotal,data6)
dftotal = rbind(dftotal,data7)

gp <- ggplot(data = dftotal, aes(x = V1), group = factor(g)) + stat_ecdf()
ggsave('cdf.eps',gp)

Does anyone knows a more efficient way to do this?


texts:
I need to create an app to predict events.

I'll explain my requirements: I've a set of intermediate events (eg.: 1, 2, 3...9) and a set of final event (eg.: A for OK and B for KO). This events bring a correlation id to relate each other.

What ML algorithm can I use to predict if a sequence of events could cause a future KO event (B in my example) or not?

I need an unsupervised algorithm.

Any idea about the algorithm?


texts: Python uses conda-base tensorflow-gpu instead of conda environment installation

texts:
I want to create a virtual environment using anaconda for python 3 in which I can use a specific version of tensorflow-gpu.

My base conda python is 3.6.8 and the base conda tensorflow-gpu version is 1.10.1.
I want to create an environment with python 3.6.9 and tensorflow-gpu 1.11.0.
I am running Ubuntu 18.04.

To do this, I have been running the following code in my bash:

$ conda create --name test_env python=3.6.9
$ conda activate test_env

However, when I open a python console and check the tensorflow version it is instantly 1.10.1 which is the tensorflow-gpu version of my conda base environment.

Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.10.1'

The problem seems to only occur with the tensorflow package. Other modules like keras give the expected ModuleNotFoundError error when I try to import them after i created the test_env environment.

Even if I install a specific tensorflow version with the following command, the tensorflow version remains to be 1.10.1.

$ conda install tensorflow-gpu==1.11.0

When i use conda list it (only) lists the target tensorflow-gpu version:
tensorflow-gpu            1.11.0               h0d30ee6_0

Do you have any ideas on how to use tensorflow-gpu version 1.11.0 in a new  virtual environment?


texts: How to capture the detail of an attribute in a sentence?

texts:
The problem is best explained using an example, so please consider the sentence below:
Made of airy cotton with a touch of stretch, the Pinafore Dress features a modern square neckline.
Here, cotton (FABRIC) and square neckline (NECKLINE) are two important attributes in the sentence. What I need to do is to capture the word airy which is a detail of the fabric. FABRIC and NECKLINE can be successfully captured using NER, but NER is not working well when it comes to capturing the detail terms like airy.
Can someone point out how I could solve this?


texts: How many epochs to run during hyperparameter search?

texts: If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run?  If I just compare after a few epochs, will that give me a good idea about how it will perform fully converged (say for example after 1000 epochs).  Is there any research papers where people have studied this question?


texts: Create top 10 index fund based on >100 stocks

texts:
Data

I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), price(USD), exchangeVolume(USD). The date frequency is daily.

Some stocks begin trading in 2014 and some in 2017 so the length of the date column is not equal for all stocks.

Goal

I want to see how an index fund would develop. The fund's strategy would be to on the first of each month buy the 10 largest stocks (based on market cap). Rebalancing occurs iff there is a "shift" in which stocks are top 10.

I want to use python.

Initial attempt

I'm familiar with pandas glob and little bit with numpy.

My sketch at solving this is the following

import all files using the glob package
create a matrices for: market cap and price. They have T rows and M columns, where M is the nr of stocks and T is the length of the longest series (circa 4 years).
Use pandas .resample('M') to go from daily to monthly data.
(ideally the column in the matrix above would have names, therefore using pandas is good since I can set it by df.columns = list_of_names)
create a returns matrix based on the price matrix, and a weigth matrix based on the market cap matrix.
return of index = diagonal(Return matrix * Weight matrix transposed)

But I'm a very inexperienced programmer so maybe this approach is stupid. I would like to see if there is a better way of solving the problem I have.


texts: Normalising time(minutes) related data with n other input variables(also dependant on time)

texts:
I am working on the GPS dataset of the football players with the energy expenditure being my output variable and other 15 variables being input/independent variables.

My objective is to forecast the energy progress of the players in the future matches if they are going to play for the full 90 minutes.

Say now I have 50 player's 150 matches data and the problem I am facing is that each player has played for different duration. Few players have played for 90 minutes in few of their matches and few have played only for 20-25 minutes in the match, so it is difficult for me to understand how to solve this problem.

sample data:

The highlighted columns in the dataset are the columns I generated. Predicted is my prediction using Multi Linear Regression for each player, which has an accuracy of 98%. I tried to add energy/time * 90 mins, but the result is not efficient. I have done feature analysis and selection to filter the important 15 variables out of the 90 variables in the actual dataset.

Solution Needed:

Day     Player     Actual_Time player    Actual_Energy_Spent    Energy_if played_for_90mins
1        A            20                        15000              XXXXX

2        A            35                        16000              XXXXX

3        A            30                        14000              XXXXX

So, If I am able to predict that value for 90 minutes from the past matches using the 15 input variables, I would be able to forecast easily for the future matches as well. I am not sure of how to achieve this.

The other input variables such as distance are also dependant on the time_played.Can this be done using any statistical model or should I be using any system dynamics in resolving this problem.

Upon doing some research, I came to know about transfer function(Activation function) and there is a function in python SciPy to do it, but I haven't learned anything about it before and don't know if that would solve this problem. Being the only Data Analyst Intern with no seniors to work with or to guide, I am helpless. Any idea or suggestion or reference material for such solving this problem would be highly helpful and much appreciated.

thanks.


texts: How to convert night image to day image?

texts:
I have set of night images which I will be using for self driving. But I want to convert those images into day images. I have developed algorithm based on day image but it is not good for night images , so I want to convert night images to day images then feed into the network. As far as I have explored image colourization techniques of grey scale image ( converting night image to black and white and then coloring it) ànd night to day style transfer using GANs. Is it possible to achieve the goal with the above two mentioned techniques or you can also suggest any other way.

Example link. :-  https://goo.gl/images/kTwnHB


texts: AdaBoost implementation and tuning for high dimensional feature space in R

texts:
I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. There exists a variety of different packages for this purpose; AdaBag, Ada and gbm. gbm() (from the gbm-package) appears to be my only available option, as stack.overflow is a problem in the others, and though it works, it is very time-consuming.
Questions:

Is there any way to overcome the stack.overflow the problem in the other packages, or have the gbm() run faster? I have tried converting the data.frame into a matrix without success.
When performing AdaBoost in gbm() (with distribution set to "AdaBoost"), An Introduction to Statistical Learning (Hastie et al.) mentions the following parameters needed for tuning:

Needs:

The total number of trees to fit.
The shrinkage parameter denoted lambda.
The number of splits in each tree, controlling the complexity of the boosted ensemble.

As the algorithm is very time consuming to perform in R, I need to find literature on what tuning parameters are within a suitable range for this kind of large feature space data, before performing cross-validation over the range to estimate the test error rate.
Any suggestions?


texts: What is the difference between upsampling and bi-linear upsampling in a CNN?

texts: I am trying to understand this paper and am unsure of what bi-linear upsampling is. Can anyone explain this at a high-level?


texts: Are there any Meta Knowledge bank available?

texts:
What resources do you use to learn meta knowledge ?
By meta knowledge, I mean generalized information that will help us take more informed decisions when working on a problem later.

Example of meta knowledge:

Lots of time series data ? Build a CNN
Limited in time and want to get quick insights from a dataset ? Try Random forests.
Continuous data and supervised learning ? Do a regression.

I thought people would have build quiz-like applications to help determine good algorithm candidates to test depending on a set of known conditions.
I found "Algorithm selection" on google and the Algorithm selection literature summary, but this doesn't summarize and dumb down the information for the larger public.

The goal is for me to learn useful information that'll allow me to pick algorithms and study them in details only when I really need them, instead of randomly trying many algorithms in hope one will give me better results.


texts: Understanding the effect of num_words of Tokenizer in Keras

texts:
Consider the following code:

from keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer(num_words = 5000)
tokenizer.fit_on_texts(texts)
print('Found %d unique words.' % len(tokenizer.word_index))

When I run this, it prints:

Found 88582 unique words.

My question is, isn't num_words the parameter that controls the number of words in the mapping dictionary known as tokenizer.word_index? Then why it still holds 88582 words when I explicitly asked it to keep only 5000 words?


texts: R: tobit with a varying censoring level (cnreg equivalent)

texts:
I have county-level data from each of the 50 states, and each state suppresses data differently. I've been looking everywhere to find a way to do a tobit-type regression in R that can account for the different censoring levels.

I've found out that in STATA you can do this with cnreg, so I'm wondering if there's an equivalent package in r, or any way to do it, maybe with censReg?


texts: Pandas: Compare two Dataframe and Groupby categorical

texts:
My Question is about pandas DataFrame, I have two DataFrame both follow the same structure.
df_1:
Index    Text     Category
0        Text 01        1
1        Text 02        1
2        Text 03        1
3        Text 04        1

df_2:
Index    Text     Category
0        Text 05        2
1        Text 02        2
2        Text 09        2
3        Text 04        2

I want to marge both for this purpose I use pd.concat(df_1, df_2) but it simply merge both files, but I want to merge in that manner.
df_merge:
Index    Text     Category
0        Text 01         1
1        Text 02       1,2
2        Text 03         1
3        Text 04       1,2
4        Text 05         2
5        Text 09         2

But I really don't know how to do that.


texts: How are the gradients of a Neural Network calculated just by matrix multiplication?

texts:
I would have expected some kind of derivative solving equation to be at work in order to back propagate the loss to each neuron. I hope my question is not too confused to answer.
In the network below, why does h_relu.T @ grad_y_pred (y hat - y) produce the gradient for w2? I understand sometimes the derivative is just something you multiply something by. Like the derivative of y = h_relu * w2 being h_relu, which doesn't make much sense to me either. I understand why the derivative of x^2 is 2x and that's about as far as my understanding of derivatives goes.

import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2


texts: Which tribe does Probabilistic Graphical Models fall under?

texts:
Pedro Domingos in  "The Master Algorithm" listed five tribes of machine learning algorithms:

Symbolists
Connectionists
Evolutionaries
Bayesians
Analogizers

Which category do probabilistic graphical models fall under?

From wikipedia (https://en.wikipedia.org/wiki/Graphical_model):

  A graphical model or probabilistic graphical model (PGM) is a
  probabilistic model for which a graph expresses the conditional
  dependence structure between random variables. They are commonly used
  in probability theory, statistics—particularly Bayesian statistics—and
  machine learning.

In that case, would it be "Bayesian"? What about something like Conditional Random Field? Is that Bayesian as well?


texts: In linear regression, we are fitting a polynomial to a set of data points. In Bishop's book of Pattern Recognition & Machine Learning, there are a few examples where the fit is a curve or a straight line. I am a bit confused if a curve is linear or not. The term linear means that the fit should be a linear function or a polynomial of degree 1 i.e., a straight line. But in many resources, examples are shown where the fit can be a polynomial of degree 3,9 etc. So, are these higher order polynomials linear?


texts: How to feed multiple asymmetric inputs to LSTM layer?

texts:
I'm trying to create an encoder-decoder architecture with an LSTM encoder. The intention is to use both the input image as well as the class label as inputs to the encoder, and to have them share the LSTM layers. Before working on my actual dataset, I want to get it working on MNIST just to see the architecture is set up right, so I'm using and input image of size (28, 28, 1) and a one-hot encoded class vector of size (10,).

The keras documentation presents multiple inputs as follows:

    main_input = Input(shape=(100,), dtype='int32', name='main_input')
    x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)
    lstm_out = LSTM(32)(x)
    auxiliary_input = Input(shape=(5,), name='aux_input')
    x = keras.layers.concatenate([lstm_out, auxiliary_input])

    x = Dense(64, activation='relu')(x)

However, this requires the two concatenated tensors to have an identical shape apart from the concatenation axis, which is not the case in my situation.

Here is my encoder so far - it obviously doesn't work for the reason mentioned above. In the latest iteration I attempted to flatten the input image and concatenate the 2d tensors. This works, but runs into the problem of LSTM requiring 3d input.

    """ LSTM encoder with 512 cells, softmax activation for signal classification,
        linear activation for extracting latent features"""
    input_img = Input(shape=self.img_shape)
    input_img = Flatten()(input_img)
    input_class = Input(shape=(self.num_classes,))

    merged = Concatenate()([input_img, input_class])
    encoder = LSTM(512, activation=LeakyReLU(alpha=0.2), return_sequences=True)(merged)
    encoder = LSTM(512, activation=LeakyReLU(alpha=0.2))(encoder)

    features = Dense(self.latent_dim, activation='linear')(encoder)
    classes = Dense(self.num_classes, activation='softmax')(encoder)

    return Model([input_img, input_class], [classes, features])

To be clear, I know I could just make separate LSTM layers for both inputs. I do not want to do that, I want them to share the layers.


texts: I am doing a forecasting for corona virus cases in NY, I have two models and not sure which one i should choose

texts: I am using exponential smoothing and using tableau for forecasting. The first model I included trend and removed seasonality and it predicted the number of cases going up but the quality I got according to tables was "Ok" (Tableau has 3 ranges to describe the quality of a model -Poor, OK and Good). The second model I removed trend and added seasonality to it, This showed the number of cases going down and the quality of the model was good according to tableau. Which model should I be using here, My professor says it's flattening the curve that's why it's going down in the second model while my teammates argue the first model should be the correct one.


texts: Optimize Yahoo Finance Code for Analysis

texts:
I am trying to analyze a number of companies using financial data I gathered from Yahoo Finance. I am also using the yfinance API to get some more details about the company using functions. Since I am trying to do this for a number of companies Each Iteration needs to be quick. Currently, 1 Company takes about 3 seconds.
Is it because of the API calls or requests?
Can I increase the below code efficiency?
import pandas as pd
import numpy as np
from pandas import ExcelWriter
import requests
import timeit
import yfinance as yf

def get_industry(stock):
    tickerdata = yf.Ticker(stock)
    return tickerdata.info['industry']

def get_symbol(symbol):
    url = "http://d.yimg.com/autoc.finance.yahoo.com/autoc?query={}&region=1&lang=en".format(symbol)
    result = requests.get(url).json()
    for x in result['ResultSet']['Result']:
        if x['symbol'] == symbol:
            return x['name']
def get_currency(stock):
    tickerdata = yf.Ticker(stock)
    return tickerdata.info['currency']

quarters = ['Q4-2020','Q3-2020','Q2-2020','Q1-2020','Q4-2019','Q3-2019','Q2-2019','Q1-2019','Q4-2018','Q3-2018','Q2-2018','Q1-2018']
start_time = timeit.default_timer()
r = []
for i in stock[:1]:
    try:
        df = pd.read_csv(i+'_quarterly_balance-sheet.csv')
        df = df.drop('ttm',axis=1,errors='ignore')
        df['name'] = df['name'].str.replace('\t','')
        df = df.iloc[:,:13].T
        df.columns = df.iloc[0]
        df = df[1:]
        df.insert(0,'Q',quarters)
        try:
            df.insert(0,'Currency',get_currency(i))
        except:
            df.insert(0,'Currency','Not Found')
        df.insert(0,'Ticker',i)
        df.insert(0,'Company',get_symbol(i))
        try:
            df.insert(0,'Industry',get_industry(i))
        except:
            df.insert(0,'Industry','Unknown')
        df = df.loc[:,df.columns.isin(['Q','Currency','Ticker','Company','Industry','TotalRevenue'])]
        r.append(df)
    except:
        continue
df = pd.concat(r)
# code you want to evaluate
elapsed = timeit.default_timer() - start_time
elapsed

3.041297 Seconds


texts: Policy Gradients vs Value function, when implemented via DQN

texts:
After studying Q-learning, Sarsa & DQN I've now discovered a term "Policy Gradients".

It's a bit unclear to me how it differs to the above approaches. Here is my understanding, please correct it:

From the moment I first encountered DQN, I always imagined DQN's input vector as only consisting of current state's features. On the output layer of DQN we have a vector of actions.  We then take the index of the highest-scored action $a$ and execute it.  It puts us into the next state $s'$.
To compute the error (how incorrectly we estimated the score of $a$) we supply $s'$ into our DQN, then discover its highest action similar to 1.  Once again, it will be on the output layer of DQN
Compute the error by bootstrapping $a$ to $a'$. The "error" vector will have zeros everywhere except at the index of that chosen action $a$

Was this in fact a "Policy Gradient", and not a Value-function approach as I thought initialy?

In that case, would the value-approach be a DQN with:

[state_features; action_one_hot_encoded_vector]  concatenation as input, and a single value on the output?

I got this impression after reading this Link

Is the basic idea of DQN Policy Gradient that simple, or am I getting things wrong?

Edit: there is a really awesome lecture about policy grads. Unfortunatelly the video is deliberately unlisted, so that normal people can't really get to it - but I am for free education, so here it is:  CS294-112 9/6/17


texts: Query related to Matlab Neural Network toolbox

texts:
If anyone has used the Neural Network toolbox in matlab, what does the two values in the performance section means? Like considering this image:

The performance section has the value 0.484 and 1.32e-07. what are these two values. Among them which is the mean squared error value. If one is the mean squared error value, then what is the other value?


texts: How to utilize measurement accuracy metadata in classifier

texts:
Given that one wants to ascribe a class to groups of measurements using a classifier model, in what way can one include information about measurement accuracy?
More specifically, is there a feature engineering strategy for including information with the intent to allow the model to weight a subset of the sample more than other subsets?
As an example, say I have have coordinate and accuracy information (lat, lon, measurement_accuracy) in a time-series. I want to make a classifier which takes a group of 2 coordinate pairs, and output a class from this group. Each sample therefore consists of 2 coordinate pairs.
The measurement accuracy could be used to indicate to the model which of these measurements should be trusted most, but does not by itself give any hint towards the class the sample belongs to.
given the 3 samples
[ # learn to trust the first coordinate set more in making a prediction
    {'lat': 11, 'lon': 40, 'acc': 5},
    {'lat': 9, 'lon': 40, 'acc': 2}
],
[ # learn to trust the second coordinate set more in making a prediction
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 4}
],
[ # learn to treat these coordinates approx. equally
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 1}
]

and assuming acc is an indication of the coordinate pair measurement accuracy I would like the model to trust the first coordinate pair more in the first sample, trust the second one more in the second sample, and in the third sample trust both coordinate pairs equally.
What kind of a representation of the data could enforce this logic, given that the data will be fed into the model in a tabular way?


texts: LSTM model for multi-step forecasting with multivariate time series

texts:
Im am trying to do a multi-step forecasting with multivariate time series, I have 9 variables (Y,X1,..X8) with 2270 samples for each variable, and I am trying to predict the future values of Y (70 future values).

I am wondering how far can i get a good accuracy? I used an lstm model but a get a very low accuracy: 15%!

I normalized my data and I am using a window size of 120.

I used the code provided by tensorflow documentation : https://www.tensorflow.org/tutorials/structured_data/time_series

my data:

my predictions:

LSTM :

    multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(25))

multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')


texts:
I have a system that manages equipments. When these equipments are faulty, they will be serviced. Imagine my dataset looks like this:

ID
Type
# of times serviced

Example Data:

|ID| Type       | #serviced |
|1 | iphone     | 1         |
|2 | iphone     | 0         |
|3 | android    | 1         |
|4 | android    | 0         |
|5 | blackberry | 0         |

What I want to do is I want to predict "of all the equipments that have not been serviced, which ones are likely to be serviced" ? (ie) identify "at risk" equipments.

The problem is my training data will be #serviced > 0. Any #serviced=0 will not be frozen and dont seem to be valid candidates to include in training. (ie) When it fails, it will be serviced hence the count will go up.

Is this a supervised or unsupervised problem ? (supervised because I have serviced and not-serviced labels, unsupervised because I want to cluster not-serviced with serviced and there by identify at-risk equipments)
What data should I include in training ?

Note:

The example is obviously simplified. In reality I have more features that describe the equipment.


texts: Loss function for optimising precision & recall / sensitivity & specificity?

texts:
I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity & specificity is what I want to optimise for. Every epoch I output it:

class SensitivitySpecificityCallback(Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch:
            x_test = self.validation_data[0]
            y_test = self.validation_data[1]
            predictions = self.model.predict(x_test)
            output_sensitivity_specificity(epoch, predictions, y_test)

def output_sensitivity_specificity(epoch, predictions, y_test):
    y_test = np.argmax(y_test, axis=-1)
    predictions = np.argmax(predictions, axis=-1)
    c = confusion_matrix(y_test, predictions)
    print('Confusion matrix:\n', c)
    print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
    print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))

On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…


texts:
I'm using Tensorflow 2 and using model.fit for training the model. but when I try to run it, it says Failed to convert a NumPy array to a Tensor (Unsupported object type float). Where I'm making mistake?enter image description here
import tensorflow as tf
print(tf.__version__)

import pandas as pd
import matplotlib.pyplot as plt

data = pd.read_csv('datatraining.txt').values

type(data)

import numpy as np

data.shape

X = data[:,2:7]
Y = data[:,7]

X.shape

Y.shape

from sklearn.model_selection import train_test_split

# split the data into train and test sets
# this lets us simulate how our model will perform in the future
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)
N, D = X_train.shape

model = tf.keras.models.Sequential([
  tf.keras.layers.Input(shape=(D,)),
  tf.keras.layers.Dense(1, activation='sigmoid')
])

# Alternatively, you can do:
# model = tf.keras.models.Sequential()
# model.add(tf.keras.layers.Dense(1, input_shape=(D,), activation='sigmoid'))

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Train the model
r = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=100)

# Evaluate the model - evaluate() returns loss and accuracy
print("Train score:", model.evaluate(X_train, y_train))
print("Test score:", model.evaluate(X_test, y_test))


texts: How does one deploy a model, after building it in Python or Matlab?

texts:
I have been playing around with a lot of different machine learning models (clustering, neural nets, etc...), but I am sort of stuck on understanding what happens after you finish building the model in Python or Matlab.

For example, let's say that I trained a basic neural network model for a binary classification problem.  How does one deploy that to colleagues, for example, so that they can load in the dataset to spit out a prediction?  I have trained a model, but what happens to that model, now?  How do I "save" that model that I just trained in Python?

I see a lot of tutorials on how to pre-process data, train the model, spit out the statistics and predictions; but, what comes next?

Obviously Facebook, Google, and anyone else heavily involved in machine learning / AI applications are creating a framework to use their models.  But, is there software that allows you to pull in data and then apply it to your Python code?  Is this what Weka, TensorFlow, and these other packages do?


texts: @RISK Vs R. Monte Carlo simulation in R?

texts:
Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks?

http://www.palisade.com/risk/

@RISK (pronounced “at risk”) performs risk analysisusing Monte Carlo simulation to show you many possible outcomes in your spreadsheet model—and tells you how likely they are to occur. It mathematically and objectively computes and tracks many different possible future scenarios, then tells you the probabilities and risks associated with each different one. This means you can judge which risks to take and which ones to avoid, allowing for the best decision making under uncertainty. (Taken from their website)

My main purposes for it would be for the construction company I currently work for:

 Able to provide more accurate models for job costing, e.g. time to construct certain parts, associated costs, etc.

 Provide more accurate models in regard to advertising and marketing, allowing us to better understand and predict what provides the better return.

If this isn't the right forum for this question, please tell me where it needs to be asked.

Thank you,


texts: I am working on a project in which I use data of movies and I represent each movie as a vector of length of 15. So there are 15 features ranging from genre to director. Most of the features are categorical and I want to use target encoding, but I am not sure if I must weight target encoded data or they are kinda already weighted. What I am trying to make is a movie recommendation system which is fully unsupervised. One hot encoding is out of question for me as one movie vector becomes easily 1 million D. Also I do not understand the reason for weighting at all. I am a rookie. Can you help me understand? By the way, categorical and numerical data(such as duration of the movie) is placed together in a vector. I hope it does not cause problems as I am planning to normalize/constrain numerical values to 0-1 range.


texts: How can I calculate AUC from the ROC curve for the classification?

texts: Based TPR and FPR, I have generate ROC curve for my binary classification model. I do not know, how to calculate AUC value. I would be very help for me if you can help me to calculate AUC value.


texts: Dividing the weights obtained on an already standardized data set by the standard deviation of the features? (Ridge regression)

texts:
I'm trying to understand a code snippet from my lecture on Machine Learning (see the code below).

It extracts the mean and standard deviation of the features and uses them to 'normalize' (standardize) the features. The data X contains all-ones in the first column. The variable t is the output value on which we train the weights.
First the mean of the output is obtained so that ridge regression can be applied without a bias term (intercept), then the all-ones column is removed and the data is normalized. The weights are obtained by the formula:

What I don't understand is that the weights obtained by using the normalized data are once again divided by the standard deviation of the features W[1:, 0] = Wn[:, 0] * Sinv. Can someone help me out? Thanks!

PS: I know normalizing is something else and I actually mean standardizing, but my professor uses normalizing instead for some reason..

Here's the code:

def NormalisedRidgeRegression(X, t, p_lambda=0.0):
    K = X.shape[1] - 1

    M = np.mean(X[:, 1:], axis=0)
    S = np.std(X[:, 1:], axis=0)
    Sinv = 1 / S

    targetmean = np.mean(t)
    tn = t - targetmean

    Xn = np.dot(X[:, 1:] - M, np.diag(Sinv))

    Wn = np.dot(np.dot(np.linalg.pinv(np.dot(Xn.T, Xn) + p_lambda * np.eye(K)), Xn.T), tn)
    W = np.zeros((K + 1, 1))
    W[1:, 0] = Wn[:, 0] * Sinv
    W[0, 0] = targetmean
    y_hat = np.dot(X, W)
    return W, y_hat


texts:
I have a dataset which is probabilistic in nature. For instance, if a record has d features, then values of $x_1$, ..., $x_d$ are such that their sum adds to 1. In other words, for each $x_i$ data record (row)

$$P(x_{i,1}) + P(x_{i,2}) + ...+ P(x_{i,d}) = 1$$

I want to apply Linear Regression Model to predict a variable $y$.

My questions are:

can Linear Regression work for such datasets? What are other models I can use?
Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).
As I use Linear Regression all independent variables
($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $\mathbf{w}$ for
$\hat{y}_i = \mathbf{w} \cdot \mathbf{x}_i$ calculated would be very large)?

Ideally, I am applying gradient descent for a convex optimization problem. It can be that $x_0...x_d$ belong to some unknown distribution and in my case, I know that distribution. But if there is an alternate model them pls suggest.


texts:
I am looking for a clustering procedure that will group a number of 3D points on the basis of their spatial relations and multivariate dimensions. Dimensions are mostly represented with (interval) metric variables and few composed of categorical variables.
My questions:

Is there any clustering procedure that also factors in weighting
coefficients assigned (by user) to certain dimensions or spatial
relation (distance)?
Is there any clustering procedure that analyses both categorical and
metric dimensions?
Is there a clustering procedure thatr fullfill these requirements
and it is already implemented in any of the commonly-used
statistical software such as     R, MATLAB, Python?

My data looks like this:

  0.0303   0.0763   0.1363    0.1753    0.1916    0.2411    0.2661
  0.0000   0.0000   0.0000    0.0084    0.0176    0.0393    0.0482
  0.3287   0.3794   0.4887    0.7151    1.0220    4.8060    7.4140
  0.2310   0.2692   0.3563    0.4384    0.4836    0.6694    0.8040

Is there a method that considers both numerical and alphabetical values? For example if I add to this dataset a column with values such as A, B, C, AC, CF,....
Thanks


texts: Different results obtained for OneVsOneClassifier (or OneVsRestClassifier) when using ordinary KFold and StratifiedKFold cross validation

texts:
When I fitted a OneVsOneClassifier (or OneVsRestClassifier), I noticed I obtained different results when I used ordinary KFold and StratifiedKFold cross validation. The testing set performance is much lower when ordinary KFold is used compared to when StratifiedKFold is used.

Questions:

1) Am I right to say for OneVsOneClassifier (or OneVsRestClassifier) strategy to work, the proportion of each class needs to be preserved in training and testing sets?

2) Is it better to use ordinary KFold or StratifiedKFold when I want to use OneVsOneClassifier (or OneVsRestClassifier)?

Many thanks.


texts: ship class Object detection with custom model weights

texts:
I have made a model to classify different categories of ships(yacht,catamaran,rubber boat) in Python and i hit a 70%  accuracy in training so now i have my weights.hdf5 file.Now i need to object detection the images i used with boundaries boxes and rectangles to find the different ships in each image, like a Yolo pretrained model but with my own weights that i trained for classes,so that the detector is going to find the ship in the image and display is's label with is's probability,
etc yacht(70.0% )rubber boat(65.0%)...
Can anyone with some more experience drive me to achieving this task?


texts: SciKit-Learn: predict values sometimes different from top predict_proba entries?

texts:
I noticed that in my text classification problem, I get significantly worse results when I use the max of predict_proba's output as compared to the straight predict value. Digging further, I found that the highest entries in each row of predict_proba don't always have index equal to the entry of predict. Example:

> [*test.target]
[61, 13, 11, 89, 11, 71, 118, 33, 52, 57, 16, 57, 100, 24, ...]
> [*text_clf.predict(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]
> [max(enumerate(prob), key=lambda p: p[1])[0] for prob in text_clf.predict_proba(test.data)]
[60, 16, 11, 16, 11, 87, 25, 32, 16, 56, 16, 115, 11, 25, ...]

You can see that predictions roughly follows test.target (not the greatest accuracy, but cest la vie), and predict_proba roughly follows predict, however for values above 25 there's an offset of 1. And for values above 87 there's an offset of 2. 115/118 and so on. Any idea why this might be? I would expect those two expressions to be identical.

My pipeline:

[
    ("vect", StemmedCountVectorizer(ngram_range=(1, 2), max_df=0.8, min_df=3),),
    ("tfidf", TfidfTransformer(use_idf=True)),
    (
        "clf",
        SGDClassifier(
            loss="modified_huber", penalty="l2", alpha=0.001, random_state=42
        ),
    ),
]


texts:
Can anybody explain Apache Kafka for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? What does messaging mean? Messaging between what?!! At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

PS:
Please explain as you would explain for a non-technical person


texts: How to initialize word-embeddings for Out of Vocabulary Word?

texts:
I am trying to use CoNLL-2003 NER (English) Dataset and I am trying to utilize pretrained embeddings for it. I am using SENNA pretrained embeddings. Now I have around 20k words in my vocabulary and out of this I have embedding available for only 9.5k words.

My current approach is to initialize an array of 20k X embedding_size with zeros and initialize the 9.5k words whose embeddings is known to me and make all the embeddings learn-able.

My question is what is the best way to do this? Any reference to such research will be very helpful ?

I have also asked a similar question later on Stackoverflow, here is the link.


texts:
In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and all output word representations. Now, we just compute the softmax over these values and try to maximise the corresponding product of context probabilities.
But shouldn't cosine similarity be a better metric than dot product for this? (Cosine similarity is normalised dot product).
So why do people argue that dot product should be high for the relevant context words, where in the reality, it should be the cosine similarity which should be high.
Also, suppose we don't use any approximation techniques for the model, and compute all the softmax values accurately. So isn't the second weight matrix as good as the first one in this case? (Normally, I have read that for obtaining the word embeddings, the first weight matrix is used, but why can't output one be used?)


texts: Why does cost function on a neural network increase?

texts:
I'm training a two layer neural network and outputting the cost function during iteration and noticed that the cost function increases dramatically with increasing iteration number. Initially I suspect it's because the learning rate of gradient descent is too high, so I changed from 0.05 to 0.005. However, this doesn't help at all. Any suggestion is highly appreciated!

Below is part of my code:

# construct model
y_pred = multilayer_perceptron(x, weights, biases)

# define cost function(mean squred error) and optimizer(gradient descent)
cost = tf.reduce_mean(tf.square(y - y_pred))
optimizer = tf.train.MomentumOptimizer(learning_rate = learning_rate,
            momentum = momentum).minimize(cost)

EPOCHS = 100
# initialize parameters
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    for epoch_no in range(EPOCHS):
        _, c = sess.run([optimizer, cost], feed_dict={x:X_train , y:Y_train})
       print('Epoch number: {}, cost: {}'.format(epoch_no, c))

# For running test dataset
results, test_cost = sess.run([y_pred, cost], feed_dict={x:X_train, y:Y_train})
print('test cost: {:.3f}'.format(test_cost))
print(y_pred)

Output for the first 3 Epoches:

Epoch number: 0, cost: 509.89886474609375
Epoch number: 1, cost: 287486752.0
Epoch number: 2, cost: 2.262859251393233e+18